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COMPUTER DATA ENTRY AND MANIPULATION 
~ ^ ' ^ APPARATUS AND METHOD \ 



/I 



BACKGROUND OB' THE INVENTION 



A 



Field of the Invention 

This invention relates generally to the field 
of devices for data entry and manipulation in computers, 
and relates more particularly to an apparatus and 
method for entering data into a computer and manipulat- 
ing virtual objects defined by the computer based on 
the gestures and positions of the hand, or other parts 
of the body, of an operator. 

Description of the Prior Art 

Input devices for computers include such 
devices as keyboards, digitizers, joysticks, mice, 
trackballs, and light pens. One function of these 
input devices is to position, in two dimensions, a 
cursor on the display screen of a computer. Once the 
cursor is positioned at a desired location, the computer 
typically will be instructed to perform an operation. 
The processes of positioning the cursor and selecting 
the operation are discrete operations, since separate 
motions are required to perform each operation. With a 
mouse, for example, cursor positioning is accomplished 
by moving the mouse along a surface, while selection of 
the operation is accomplished by pushing keys located 
either on the mouse or on a separate keyboard. Mastering 
the operation of such input devices is often difficult 
because the hand movements required to operate the 
devices do not correspond to the visual feedback 
presented by the display screen of the computer. 
Furthermore, the operator's hand(s) must be removed 



from the keyboard, positioned on the mouse, then 
returned to the keyboard. 

Glove input devices also have been used to 
supply data to computers. U.S. Patent No. 4,414,537, 
filed September 15, 1981, by G. Grimes and entitled 
"Digital Data Entry Glove Interface," describes one 
such glove input device. The Grimes patent discloses a 
glove with sensors for detecting the flexing of finger 
joints, sensors for detecting contact between various 
portions of the hand, and sensors for detecting the 
orientation of the hand. The Grimes device is used to 
identify static hand positions representing the charac- 
ters of the alphabet. Furthermore, the glove is 
designed to differentiate from one another a fixed 
number of static shapes representing the letters of the 
alphabet . 



SUMMARY OF THE INVENTION 



The present invention provides an apparatus 
and method for manipulating virtual objects defined by 
a computer according to the gestures, position, and 
movement of the hand of an operator. Such manipulation 
includes positioning a cursor or other representation 
of the hand of the operator with respect to virtual 
objects defined by the computer. Operations on those 
virtual objects may then be carried out according to 
certain gesture specifying movements of the operator's 
hand. The virtual objects themselves may be represen- 
tations of computer input devices such as a joystick, 
mouse, pen, keyboard, paintbrush or other devices. 
More generally, the objects may be tools which themselves 
act on other virtual objects. For example, a virtual 
steering wheel may be used to drive a simulation of an 
automobile, or to provide input to a remote system. 

The invention includes gesture sensing means 
coupled to the hand for detecting gesture specifying 



movements of the hand, such as flexing of the fingers, 
as well as hand position sensing means for detecting 
the position of the hand with respect to the display. 
Signal processing means are provided to receive data 
from the gesture sensing means and the hand position 
sensing means to instruct the computer to manipulate 
the cursor and/or virtual objects according to the 
movements of the operator's hand. 

In one embodiment of the present invention, 
the gesture sensing means includes a glove assembly 
with attached sensors that are responsive to the degree 
of flex of the fingers of the operator's hand. These 
flex sensors are mounted on a flexible printed circuit 
board and are sandwiched between an inner and an outer 
glove. A decoding circuit for addressing the sensors 
is also mounted on the flexible printed circuit board, 
and is electrically coupled to the sensors through the 
flexible printed circuit board and to the computer via 
a detachable cable. The hand position sensing means 
preferably includes one or more ultrasonic transmitters 
affixed to the glove assembly, a stationary receiver 
comprising three separate spaced-apart ultrasonic 
receiving units, and a control circuit that measures 
the time delay of pulsed ultrasonic signals from the 
transmitter to the three receivers. The time delay 
provides a measure of the spatial position of the 
operator's hand. The signal processing means includes 
interface circuitry for coupling the glove to the host 
computer, for positioning a hand-shaped cursor on the 
display screen of the computer according to the position 
of the operator's hand, for responding to output 
signals from the flex sensors, and for manipulating 
virtual objects defined by the computer according to 
commands represented by the gestures and movement of 
the operator's hand. A database within the host 
computer can be employed to provide constraints, such 



as inertia, linkage to other objects, etc., for the 
objects being manipulated. 

The present invention also comprises a 
computer data entry and manipulation apparatus and 
method capable of determining the dynamic gestures of 
an operator's hand and the spatial position of the 
hand. As an input device, the present invention is 
especially well adapted for use with a pictorial or 
symbolic programming language having a dynamic cursor 
which corresponds in shape to the shape of the glove 
and moves on the screen in response to movement of the 
glove in space. The present invention provides a basis 
for use of a symbolic programming language in which the 
physical gestures of the operator's hand are used to 
implement conceptually similar and easily recognizable 
functions or operations on virtual objects displayed on 
the display screen of the computer. 

BRIEF DESCRIPTION OF THE DRAWINGS 



Figure 1 is a perspective view of a preferred 
embodiment of the computer data entry and manipulation 
apparatus of the inventia 

Figure ^is a perspective view of the glove 
assembly 12 shown in Figure 1, including sensors and 
and ultrasonic transmitter; 

Figure 3>*<ls a circuit schematic of a preferred 
embodiment of the circuitry on glove 12; 

Figure 4 is a circuit schematic of an ultrasonic 

receiver; 

Figure 5 L^^a schematic of an interface 



circuit used to connect a preferred embodiment of the 
apparatus of the invention to a host computer; and 

Figure 6 is a flowchart of software employed 
to sense when a virtual object is picked up. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



Figures 1 through 6 of the drawings depict 
various preferred embodiments of the present invention 
for purposes of illustration only. One skilled in the 
art will recognize from the following discussion that 
alternative embodiments of the structures and methods 
illustrated herein may be employed without departing 
from the invention. 

In Figure 1, a preferred embodiment of the 
present invention is illustrated in its intended mode 
of use, namely as a computer data entry and manipulation 
apparatus 10. The apparatus 10 includes a glove 
assembly 12 electrically coupled via cable 13 to an 
interface circuit 14 that is, in turn, connected to a 
port of a host computer 16. A position sensing receiver 
assembly 20 consisting of three receivers disposed 
around the screen 28 is also electrically coupled to 
the interface circuit 14. As explained below, the 
glove assembly 12 contains sensors that detect the 
flexing of the fingers and other gestures of the hand 
of an operator, and also contains one or more ultrasonic 
transducers 17 for transmitting signals to receivers 20 
to enable detecting the spatial position of the glove 
assembly 12 with respect to the computer display. The 
position sensing receiver assembly 20 includes three 
ultrasonic receivers 24 located at corners of display 
28 facing toward the operator. 

In operation, the glove assembly 12 is worn 
on the hand of an operator, and is used to position a 
cursor 26, typically a representation of the glove 12, 
on the display screen 28 of the computer 16. A computer 
generated virtual object is displayed on the screen 28. 
The spatial position of the glove assembly 12 is 
determined by the time delay between transmission of an 
ultrasonic signal by transducer 17 and the reception of 
that signal by the receivers 20 of the position sensing 
receiver assembly 20. The position and orientation of 



the fingers is transmitted to the interface circuit 14 
by conductive cable 13, although other well-known 
techniques such as radio could be employed. Software 
within the host computer 16 converts the time delay 
data into orthogonal coordinates, and directs the 
computer 16 to display the cursor 26 on the display 
screen 28 accordingly- Thus, movement by the glove 
assembly 12 in a plane parallel to that of the display 
screen 28 results in corresponding movement by the 
cursor 26. Movement by the glove assembly 12 toward 
and away from the display screen 28 can be represented 
by varying the size of the glove representation cursor 
26 . 

Signals from the glove assembly 12 may also 
enter commands into the computer 16. As described in 
detail below, glove assembly 12 contains sensors that 
respond to the gestures of the operator's hand. The 
software receives and interprets gesture indicating 
data from the sensors of the glove assembly 12 and 
enters commands into the computer 16 according to the 
gestures recognized. These commands relate to the 
manipulation of virtual objects created by the computer 
16 and displayed on the display screen 28. For example, 
Figure 6 is a flow chart illustrating software for 
sensing when an object has been "picked up." 

Figure 2 illustrates the preferred embodiment 
of the glove assembly 12. An outer glove, not shown, 
protects the circuitry attached to the inner glove. 
The component parts of the glove assembly 12 are bonded 
or otherwise secured to an inner glove 32, which is 
worn on the hand of the operator during operation. In 
the illustrated embodiment, the sensors and electrical 
components of the glove assembly are soldered to and 
electrically interconnected by a flexible printed 
circuit board 34 (Figure 2) , which is itself bonded or 
otherwise secured to the inner glove 32. The flexible 
printed circuit board 34 includes five elongated 
portions 36 positioned along the back side of the 



fingers and thumb and extending from a central portion 
38 positioned along the back of the hand. Preferably, 
the inner glove 32 is formed from a material such as 
stretch knitted nylon which accommodates various sizes 
of hands and maintains a snug fit during use. The 
outer glove covers and protects the components of the 
glove assembly 12, and improves the aesthetics of the 
glove assembly. Preferably, the outer glove (not 
shown) is composed of a light weight and durable 
material such as cotton. 

The glove assembly 12 includes flex sensors 
40, each positioned on the back side of the inner glove 
32 disposed along the fingers to measure bending. The 
flex sensors 40 are preferably of the type that will 
provide a signal that is an analog representation of 
the degree of bend of each of the fingers and thumb. 
The flex sensor 40 comprises a flexible tube 42 having 
interior reflective walls with a light source 44 at one 
end and a photosensitive detector 46 at the other end. 
The light source 44 is preferably an infrared light 
emitting diode, and the photosensitive detector 46 is 
preferably a phototransistor . The tubing is preferably 
black glossy soft vinyl. The flexible tube 42 is 
bonded or otherwise secured to the flexible printed 
circuit board 34, with the electrical leads of the 
light source 44 and the photosensitive detector 46 
soldered to appropriate conductive traces of the 
flexible printed circuit board. The amount of light 
that impinges on the photosensitive detector 46., and 
the corresponding amount of current flowing through the 
photosensitive detector, is dependent upon the amount 
of bend of the flexible tube 42. When the finger is 
extended the flexible tube 42 is generally straight and 
a maximum amount of light from the light source 44 
impinges on the photosensitive detector 46. As the 
finger is flexed progressively, the portion of the 
tube's 42 reflective inner wall that is mutually viewed 
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by both the light source 44 and the photosensitive 
detector 46 decreases, which restricts the amount of 
light transmitted to the photosensitive detector 46. 
Thus, the flex sensors 40 provide an analog signal that 
indicates the flexing of the operator's fingers and 
thumb. A detailed description of typical flex sensors 
may be found in U.S. Patent No. 4,542,291, entitled 
"Optical Flex Sensor," which is commonly assigned. 

The glove assembly 12 includes circuitry 
which is described below in conjunction with Figure 3. 
The decoder chip 48 is soldered to appropriate conductive 
traces at the central portion area 38 of the flexible 
printed circuit board 34. The glove assembly 12 is 
electrically connected to the interface circuit 14 via 
a cable 13, which is preferably a flat ribbon cable 
that is releasably attached to the flexible printed 
circuit board 34 by a connector 54. The cable 13 also 
supplies power and ground signals to the components of 
the glove assembly 12. An optical, radio or other 
electromagnetic transmitter could also be employed. 

As mentioned above, the glove assembly 12 
includes at least one ultrasonic transducer 17, for 
example, a high frequency tweeter, that transmits 
ultrasonic signals for use in determining the spatial 
position of the glove assembly. Two transducers are 
used in some embodiments to provide roll and yaw hand 
orientation information, and so that at least one will 
be within " line-of-sight" of the receivers. The 
ultrasonic transducer ( s) 17 is soldered to appropriate 
conductive traces at the central portion area 38 of the 
flexible printed circuit board 34, and is electrically 
connected to the interface electronics 14 via the cable 
13. Preferably, components 55, including a transformer 
and transistor for controlling the ultrasonic transducer 
17, are also contained in the glove assembly 12 and 
mounted to the flexible printed circuit board 34. 



In some embodiments, to counteract possible 
blocking of the ultrasonic signals by the operator's 
hand, a flexible transmission tube 56 is utilized to 
conduct the ultrasonic signals to a different part of 
the glove assembly. Transmission tube 56 may extend in 
any desired direction from the ultrasonic transducer 17 
to prevent shadowing. The transmission tube carries 
the ultrasonic signals transmitted by the ultrasonic 
transducer 17 and radiates those signals out an open 
end. The transmission tube ensures that hand gestures 
do not block the transmission of the ultrasonic signals 
to the position sensing receiver assembly 18. 

In addition to the flex sensors, the glove 
assembly 12 may also include a hand orientation sensor 
70 which provides data indicative of the orientation of 
the glove assembly relative to the three rotational 
axis of roll, pitch, and yaw. The orientation sensor 
70 can be implemented in various ways, such as a 
three-axis accelerometer , an array of mercury potentio- 
meters, or a bubble gauge read electro optically. In 
the preferred embodiment, low frequency magnetic fields 
like the 3SPACE™ system are employed. This system is 
available from the Polhemus Navigation Sciences Division 
of McDonnell Douglas Electronics Co., Essex Junction, 
Vermont . 

As an alternative to the use of spatial 
positioning of the glove assembly 12 for directing the 
two-dimensional positioning of the screen cursor 26, 
wrist motions may be used. For example, the forward 
and back flexing of the wrist can indicate vertical 
positioning of the screen cursor 26, while left and 
right flexing of the wrist can indicate horizontal 
positioning of the screen cursor. To achieve this 
additional flex sensors may be secured to the inner 
glove at locations surrounding the wrist joint. 

Figure 3 is a schematic illustrating the 
circuitry present on the glove according to a preferred 
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embodiment of the invention. The circuitry shown 
includes a series of light-emitting diodes LXO , I/YO ... LX7 , 
LY7 together with a series of solid state photodetector s 
PXO , PYO . . . PX7 , PY7 . As explained above, these light- 
emitting and sensing devices are employed in the bend 
sensors disposed on the glove, and provide direct 
readings of finger bending. The circuitry shown also 
includes a dual 2-line to .A^line demultiplexer Ul f 
typically a 74L S 156 integrated circuit. This circuit 
receives signals from the host computer and sequentially 
turns on the appropriate light source by connecting one 
of terminals L0-L7 to ground. The output signal from 
the photodetector is supplied on one of lines Jl.5,J1.9 
to the detector circuitry, which is described below. 

The ultrasonic transmitter XTl is controlled 
by a transistor Ql , preferably a type 2N2222 under 
control of signals supplied on line Jl.4, which are 
supplied through a 10 : 1 step-up transformer Tl . When 
the transmitter is turned on, a pulse of high frequency 
(about 40 kHz) sound is broadcast. The sound is 
received by the ultrasonic receivers 20 disposed around 
the monitor. Preferably, the ultrasonic transmitter 
XTl is an ultrasonic piezoelectric ceramic tweeter, 
while the ultrasonic receiver comprises an ultrasonic 
piezoelectric ceramic microphone. Both such devices 
are commercially available from Panasonic. The time 
delay between the transmission of the signal by trans- 
mitter XTl and its reception by the receivers 20 is 
indicative of the distance between the transmitter and 
each of the receivers. (Ultrasonic sound travels at 
about 330 meters per second.) The three distances 
measured between the glove and the three receivers 
define the position of the hand with respect to the 
receivers . 

If PA is the distance from the ultrasonic 
transmitter to receiver A, PB the distance to receiver 
B and PC the distance to receiver C, and if AB and AC 



equal the distance between A and B, and A and C, 
respectively , then 



x = PA2 - PB2 + AB2 . A _ pB + ^ 



?C2 - \f Q + AC2 - K 2 (PC - PA + ^AC) 



z = "\/pA 2 - X 2 - Y 2 ' = K 0 (PA + PB + PC) 



where x, y and z are the distances from the origin of a 
rectangular coordinate system. Because the position 
equations require squares and square roots, the approxi- 
mations are easier to calculate, requiring only addition 
subtraction and scaling by constants K^, t and . 

The measurement of this distance is initiated 
by the host computer sending a control byte to select a 
receiver 2 0 to generate a short duration high frequency 
pulse by triggering U4 in Figure 5 . The host then 
measures the time , using a counter internal to the 
computer, required for the reception of an ultrasonic 
signal. When received, this signal generates an 
interrupt by U32. The time between sending the control 
byte and the interrupt is proportional to the distance 
for the receiver 20 selected. By polling all three 
receivers sequentially at a sufficiently high 
frequency, the position of the glove with respect to 
the receivers may be readily determined. 

Figure 4 is a schematic illustrating the 
ultrasonic receivers and the interconnections among 
them. As shown in the upper portion of Figure 4, each 
receiver includes an ultrasonic receiver XRl for 
receiving the signals, together with a two-stage 



amplifier comprising two transistors Ql and Q2 , preferably 
type 2N2222, which amplify the received signal and 
supply it on line RCV. The two-stage amplifiers are 
employed immediately next to the receiver transducer to 
minimize noise. Each of the receivers is disabled 
during and shortly after the transmission pulse by Ql 
to prevent false detections. Preferably, the transmission 
lasts about_i J i v msec. (abo ut 2 0 cycles of a kHz 
signal) , while the post transmission blanking lasts for 

about 0.2 msec . 

Figure 5 is a detailed schematic of the 
interface circuitry which couples both the glove and 
ultrasonic receivers to the host computer. The interface 
circuitry includes provision for two gloves so the 
operator may wear one on each hand, if necessary. In a 
preferred embodiment the interface circuitry adapts 
between a user port, for example, on a Commodore 64 
computer, having the characteristics shown in Figure 5. 
The circuitry shown includes three jacks, Jl , J2 , and 
J3. The output signals from glove #1 shown in Figure 3 
are connected to jack Jl , while those from an optional 
glove #2 are connected into jack J2. The output 
terminals of the three receiver 20 amplifiers are 
connected into jack J3. The three jacks are connected 
to the input terminals of a dual four-channel analog 
multiplexer U2 , preferably a type 4052 integrated 
circuit. This integrated circuit functions as a two 
pole four-throw switch and routes one of the channels 
from glove #1, glove #2, or the ultrasonic receivers to 
the detector circuitry. The channel routed to the 
detector circuitry is under control of terminals Ml and 
M2 from the user port. 

The jacks Jl , J2, and J3 are coupled to 
terminals X0...X3, Y0...Y3 of switch U2 . The determina- 
tion of which of these input terminals is coupled to 
the output terminals X and Y is made under the control 
of input signals Ml and M2 connected to terminals A and 



B of switch U2 . Ml and M2 are signals supplied by the 
user port from the host computer. Under control of Ml 
and M2 , one of the Y input signals will be supplied to 
the Y output terminal or one of the X input signals 
supplied to the X output terminal. The Y terminal 
provides bend sensor information, while the X terminal 
provides information from one of the ultrasonic receivers. 

Assuming that the Y terminal of U2 has been 
selected, the host computer pulls node /C low to 
discharge capacitor C13. Once node /C is released, the 
bend sensor current supplied at output terminal Y will 
begin charging capacitor C13. Comparator U31 compares 
a reference voltage Vr with the potential on the Y 
output terminal. When the Y output terminal exceeds 
the reference voltage, which is approximately 2/3 of 
5 volts, the output of comparator U31 goes to ground 
potential. This drop in potential on node /F is 
interpreted by the host computer as an interrupt 
signal. The delay between pulling node /C low and node 
/F going low is indicative of the extent of bending of 
the bend sensor interrogated. 

The X output terminal from switch U2 provides 
information with respect to the time delay between 
transmission of the ultrasonic signal and its reception, 
and thereby information relating to the distance 
between the transmitter and the selected receiver. The 
ultrasonic pulse is generated by the circuitry in the 
lower portion of Figure 5 explained below. In the 
similar manner to that described above, the output 
signal on terminal X of switch U2 is compared by 
comparator U32 to a reference voltage. Because the 
output signal on node X is an AC signal, diode D3 
prevents it from becoming too negative. Transistor Ql 
functions to turn off the detector circuit when node T 
is held high, while a pulse is being transmitted. 
Capacitor C8 and resistor R15 provide a time delay to 
blank operation of the detector immediately after pulse 
transmission to prevent interpretation of electric 



noise as indicative of the transmitter-receiver separa- 
tion. The potentiometer R4 allows adjusting the 
reference potential to trim to allow for manufacturing 
tolerances. Once output signal X exceeds the reference 
voltage, the output node of comparator U32 is pulled 
low to create an interrupt signal on line /F, which is 
supplied to the host computer. 

The circuitry of the remaining portion of 
Figure 5 is used to generate the ultrasonic pulse for 
transmission by the transducer mounted on the glove. 
The heart of this circuit is a dual timer U4 , typically 
a type 556 integrated circuit. Half the timer determines 
the transmission duration, typically about 0.5 msec, 
while the other half determines the transmission 
frequency, typically_40 kHz. Diodes D5-D8 AND the 
resulting pulse train with the transmit enable control 
bits XI and X2 and are applied to comparators U33 and 
U34. These comparators drive a 1 : 10 step-up pulse 
transformer Tl through Ql shown in Figure 3. Potentio- 
meter R17 is employed to set the frequency and potentio- 
meter R19 controls the number of pulses transmitted. 
Bl and B2 are connected to jacks Jl and J2 and thereby 
provide the pulse train to the base of transistor Ql in 
Figure 10. 

Figure 6 is a flow chart which illustrates 
one technique by which the data entry and manipulation 
application of this invention may operate. Assume that 
an object is displayed on the screen and represented 
within the memory of the computer system by virtue of a 
table in a database indicative of the exterior appearance 
of the object. As the user of the system moves his 
hand in space, the position and orientation of the 
glove are continuously detected by the computer system 
as shown by block 140. After each determination of the 
position of the glove, the display 28 is updated to 
reflect the new glove position and orientation, as 
indicated by block 142. After the display is updated, 
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the position and orientation of the glove are checked 
to determine whether an object has been "picked up" on 
^the screen* This may be achieved using any desired 
well Jcnown algorithm, for example, by determining 
5 whether at least two points on the representation of 

the glove are coincident with at least two points on 
^tfie object. The step is shown by block 144. If no 
coincidence occurs between the glove representation and 
object, control is returned to block 140 and the 

10 position of the glove is again read. If coincidence 

between the object and glove are found, control passes 
to block 146 and the object is displayed in its new 
position. Once the object has been redisplayed, 
control returns to block 140 so that the position of 

15 the glove may be again read. 

The foregoing is a description of a preferred 
embodiment of the invention. It should be understood 
that specific details, such as component types, have 
been provided to explain the construction of the 

20 invention. The scope of the invention may be determined 

from the appended claims. 
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